Random Forest Regression Trading Model

Intent:

This modeling process was meant to test techniques that approximate a target price of multiple stocks in the S&P 500 at some point in the future. The intent is to include technical indicators, which are used by traders to uncover trends into future stock performance.

Hypothesis:

Given traders frequently make decisions on purchasing assets based on technical indicators, or statistics about the movement of stock prices and trading activity, I hopthesize that these decisions can be modeled using advanced modeling techniques and hopefully identified before actual price movement happens. Automating this via machine learning should allow us to mechanically trade faster than a normal human using manual techniques.

# Import useful packages
import os
import time
import pandas as pd
import numpy as np
from datetime import datetime, date, time, timedelta

# Suppressing some warnings in pandas
pd.options.mode.chained_assignment = None

# Model creation packages
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import make_scorer, mean_squared_error, accuracy_score, max_error
from sklearn.model_selection import train_test_split, GridSearchCV
import seaborn as sns

# Importing our self-created functions
from feature_creation import *
from portfolio import *
import port_2 as port2
import financial_metrics as fm
from port_charts import *

# Importing packages needed to reload self-created functions when testing
import sys, importlib

# Dash modules
import dash
import dash_table
import dash_core_components as dcc
import dash_html_components as html
from jupyter_dash import JupyterDash
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots

# For loading data from postgres
from sqlalchemy import create_engine
import psycopg2
from psycopg2 import connect

# Importing Financial packages used for building features and visualizations
from yahoo_fin import stock_info as si
from ta import add_all_ta_features

def import_technical_features():
    conn = connect(dbname = '697_temp', user = 'postgres', host = 'databasesec.cvhiyxfodl3e.us-east-2.rds.amazonaws.com', password = 'poRter!5067')
    cur = conn.cursor()
    query = "SELECT * FROM technical_features_daily"
    data = pd.read_sql_query(query,conn)
    data = data.sort_values(['ticker', 'Date'])
    data['Date'] = pd.to_datetime(data['Date'])
    data = data.set_index('Date')
    return data
feature_df = import_technical_features()

# Or just load if no need to re-create features
# feature_df = pd.read_csv('assets/models/tyler_rf_daily_update/ta_feature_df.csv',index_col=0)
# Ensure index is datetime
# feature_df.index = pd.to_datetime(feature_df.index)

feature_df.head()

	Adj Close	Close	Close_adj	Dividends	High	High_adj	Low	Low_adj	Open	Open_adj	...	momentum_wr	momentum_ao	momentum_kama	momentum_roc	momentum_ppo	momentum_ppo_signal	momentum_ppo_hist	others_dr	others_dlr	others_cr
Date
2016-01-04	38.763378	38.763378	40.689999	0.0	39.239704	41.189999	38.429952	40.340000	39.115862	41.060001	...	-65.066997	1.211714	38.681657	-0.021380	-18.799989	-18.876864	0.076875	-2.678761	-2.715293	42.848616
2016-01-05	38.629997	38.629997	40.549999	0.0	39.011059	40.950001	38.429941	40.340000	38.801475	40.730000	...	-75.796411	0.911207	38.680692	-1.856453	-15.666102	-18.234711	2.568610	-0.344090	-0.344683	42.357088
2016-01-06	38.801476	38.801476	40.730000	0.0	39.049167	40.990002	38.153673	40.049999	38.334679	40.240002	...	-68.275858	0.512847	38.681221	0.150508	-14.771354	-17.542040	2.770686	0.443899	0.442917	42.989010
2016-01-07	37.153393	37.153393	39.000000	0.0	38.248943	40.150002	36.972390	38.810001	38.239414	40.139999	...	-94.654171	-0.040862	38.546475	-2.477167	-8.998514	-15.833335	6.834820	-4.247474	-4.340318	36.915589
2016-01-08	36.762798	36.762798	38.590000	0.0	37.829766	39.709999	36.648481	38.470001	37.362969	39.220001	...	-96.918487	-0.611387	38.244446	-5.227764	-3.858596	-13.438387	9.579791	-1.051302	-1.056868	35.476192

5 rows × 103 columns

# Doing some data-preprocessing, as some stocks do not have adjusted features

# Filling in technical indicators with known values
feature_df = feature_df.fillna(method="ffill")
feature_df = feature_df.fillna(method="bfill")

# Some Tickers do not have adjusted prices, so setting to non-adjusted
feature_df['Close_adj'] = feature_df['Close_adj'].fillna(feature_df['Close'])
feature_df['High_adj'] = feature_df['High_adj'].fillna(feature_df['High'])
feature_df['Low_adj'] = feature_df['Low_adj'].fillna(feature_df['Low'])
feature_df['Open_adj'] = feature_df['Open_adj'].fillna(feature_df['Open'])

# Simple visualization of a ticker price over time to ensure data looks accurate
fig = go.Figure()

ticker = 'AAPL'
df = feature_df[(feature_df['ticker']==ticker)&(feature_df.index>='2016-01-01')]

fig.add_trace(go.Scatter(x=df.index,
                         y=df['Close'],
                        line={"color": "#228B22"},
                        mode="lines",
                        name='Closing Price'))

fig.add_trace(go.Scatter(x=df.index,
                         y=df['trend_sma_fast'],
                        line={"color": "red","dash":"dash","width":1},
                        mode="lines",
                        name='Closing Price'))

fig.add_trace(go.Scatter(x=df.index,
                         y=df['trend_sma_slow'],
                        line={"color": "black","dash":"dash","width":1},
                        mode="lines",
                        name='Closing Price'))

fig.update_layout(title_text=f'{ticker} Closing Price',title_x=0.5,
                         template="ggplot2",font=dict(size=10,color='black'),xaxis_showgrid=False,
                         paper_bgcolor='rgba(0,0,0,0)',
                         yaxis_title="Closing Price",margin={"r": 20, "t": 35, "l": 20, "b": 10},
                         showlegend=False)

fig.show()

Part 1: Feature Selection

In this section, we train a lightly configured Random Forest Regressor to get feature importance of all features. Using this set of features, we order based on importance and only select features over a certain threshold of importance

# Creating sample df for AAPL only, but we train models for all tickers eventually
sample_df = feature_df[(feature_df['ticker']=='AAPL')&(feature_df.index>='2018-01-01')&\
                       (feature_df.index<='2021-07-01')]

# Creating target variables to look at performance in forecasting at different time horizons
sample_df['target_7'] = sample_df['Close_adj'].shift(-7)
sample_df['target_30'] = sample_df['Close_adj'].shift(-30)
sample_df['target_60'] = sample_df['Close_adj'].shift(-60)
sample_df['target_120'] = sample_df['Close_adj'].shift(-120)

# Test Train split, ensuring data looks correct
split_perc = .8
train_df = sample_df.iloc[:int(len(sample_df)*split_perc)]
test_df = sample_df.iloc[int(len(sample_df)*split_perc):]

# Ensuring the test/train split worked correctly
fig = go.Figure()

fig.add_trace(go.Scatter(x=train_df.index,
                         y=train_df['Close_adj'],
                        line={"color": "#228B22"},
                        mode="lines",
                        name='Train'))

fig.add_trace(go.Scatter(x=test_df.index,
                         y=test_df['Close_adj'],
                        line={"color": "red"},
                        mode="lines",
                        name='Test'))

# Get actual feature columns from train_df
features = list(train_df.columns)[15:len(train_df.columns)-4]
features.extend(['Adj Close','High_adj','Low_adj','Open_adj'])

# Training our initial RF Reg model
X = train_df[features]
y = train_df['target_7']

# If scaling is needed, but not needed
# scaler = StandardScaler()
# X_Scaled = scaler.fit_transform(X)

# Using a Random Forest Regressor to test time horizon predictions
regr = RandomForestRegressor(max_depth=5, random_state=0)

regr.fit(X, y)

RandomForestRegressor(max_depth=5, random_state=0)

# Plotting feature importance. Vast majority of features seem to be of no use to the model, so will drop
fig = go.Figure()

feats = [x for _, x in sorted(zip(regr.feature_importances_,features),reverse=True)]
importance = sorted(regr.feature_importances_,reverse=True)

fig.add_trace(go.Bar(x=feats, y=importance,name="Feature Importance"))

fig.add_trace(go.Scatter(x=feats,
                         y=np.cumsum(sorted(regr.feature_importances_,reverse=True)),
                         line={"color":"black"},
                         mode="lines",
                         name="Cumulative Importance"
                        ))

fig.show()

# Getting only features that are important up to a threshold, and directionally adding ~2%
thresh = .9
thresh_len = len([x for x in np.cumsum(sorted(regr.feature_importances_,reverse=True)) if x <= thresh])

# Using this loop to ensure features we keep are adding substantial value
[x for x in np.cumsum(sorted(regr.feature_importances_,reverse=True)) if x <= thresh]

[0.422571026136333,
 0.5219121706941748,
 0.6068063257529748,
 0.682994820910156,
 0.723146840909082,
 0.7571800767638945,
 0.7830866594590546,
 0.8082678713828676,
 0.8275077193311142,
 0.8455676623182647,
 0.8629832846339253,
 0.8795663037780871,
 0.8961197372291583]

Part 2: Exploring Target Variables

We are going to explore model prediction capabilities by training 4 models tasked with predicting future price over some time horizon (7, 30, 60, 120 days), which will tell us if we should be buying or selling to take advantage of price changes

# Including all features under importance threshold
updated_feats = feats[:thresh_len+1]

# Re-training model with smaller subset of features
regr = RandomForestRegressor(max_depth=10, n_estimators= 100, random_state=0)

X = train_df[updated_feats]
y = train_df['target_7']

# If scaling is needed
# X_Scaled = scaler.fit_transform(X)

regr.fit(X, y)

feat_importance_7 = regr.feature_importances_
r2_7 = np.round(regr.score(X,y),4)

# R^2 is very high. This model is most likely highly overfit
print(f'R^2 for Model: {np.round(regr.score(X,y),4)}')

R^2 for Model: 0.9992

# Creating our train data points from predicting our target
train_predictions = regr.predict(X)

X_test = test_df[updated_feats]

# If scaling is needed
# X_test_scaled = scaler.fit_transform(test_df[updated_feats])

# Creating our test data points from predicting our target
test_predictions = regr.predict(X_test)

fig = go.Figure()

fig.add_trace(go.Scatter(x=train_df.index,
                         y=train_df['Close_adj'],
                        line={"color": "#228B22"},
                        mode="lines",
                        name='Train'))

fig.add_trace(go.Scatter(x=train_df.index+timedelta(days=7),
                         y=train_predictions,
                        line={"color": "gray","dash":"dash"},
                        mode="lines",
                        name='Train Predictions'))

fig.add_trace(go.Scatter(x=test_df.index,
                         y=test_df['Close_adj'],
                        line={"color": "#228B22","dash":"dash"},
                        mode="lines",
                        name='Test Actuals'))

fig.add_trace(go.Scatter(x=test_df.index+timedelta(days=7),
                         y=test_predictions,
                        line={"color": "red"},
                        mode="lines",
                        name='Test Predictions (7 Day)'))

Observations:

1. Looking at prediction performance. So far, our model does a horrible job at predicting the future

Adding additional models with longer targets

# Training a model with 30 day target
regr = RandomForestRegressor(max_depth=5, random_state=0)

X = train_df[updated_feats]
y = train_df['target_30']

regr.fit(X, y)

feat_importance_30 = regr.feature_importances_
r2_30 = np.round(regr.score(X,y),4)

print(f'R^2 for Model: {np.round(regr.score(X,y),4)}')

test_predictions = regr.predict(test_df[updated_feats])

fig.add_trace(go.Scatter(x=test_df.index+timedelta(days=30),
                         y=test_predictions,
                        line={"color": "red","dash":"dash"},
                        mode="lines",
                        name='Test Predictions (30 Day)'))

R^2 for Model: 0.9917

# Training a model with 60 day target
regr = RandomForestRegressor(max_depth=5, random_state=0)

X = train_df[updated_feats]
y = train_df['target_60']

regr.fit(X, y)

feat_importance_60 = regr.feature_importances_
r2_60 = np.round(regr.score(X,y),4)

# R^2 is very high. This model is most likely highly overfit
print(f'R^2 for Model: {np.round(regr.score(X,y),4)}')

test_predictions = regr.predict(test_df[updated_feats])

fig.add_trace(go.Scatter(x=test_df.index+timedelta(days=60),
                         y=test_predictions,
                        line={"color": "blue"},
                        mode="lines",
                        name='Test Predictions (60 Day)'))

R^2 for Model: 0.9937

# Training a model with 120 day target
regr = RandomForestRegressor(max_depth=5, random_state=0)

X = train_df[updated_feats]
y = train_df['target_120']

regr.fit(X, y)

feat_importance_120 = regr.feature_importances_
r2_120 = np.round(regr.score(X,y),4)

# R^2 is very high. This model is most likely highly overfit
print(f'R^2 for Model: {np.round(regr.score(X,y),4)}')

test_predictions = regr.predict(test_df[updated_feats])

fig.add_trace(go.Scatter(x=test_df.index+timedelta(days=120),
                         y=test_predictions,
                        line={"color": "blue","dash":"dash"},
                        mode="lines",
                        name='Test Predictions (120 Day)'))

R^2 for Model: 0.994

# Looking at correlation between features, we have quite high correlation between all
corr = train_df[updated_feats].corr()

sns.heatmap(corr)

<AxesSubplot:>

# Inspecting features by importance
[x for _, x in sorted(zip(feat_importance_7,updated_feats),reverse=True)]

['Low_adj',
 'Adj Close',
 'Open_adj',
 'others_cr',
 'volume_vwap',
 'volume_nvi',
 'volatility_kcl',
 'volatility_bbl',
 'trend_visual_ichimoku_a',
 'trend_ichimoku_base',
 'volatility_dcl',
 'trend_psar_up',
 'lowerband',
 'High_adj']

Quick Observations for Correlation:

1. Fairly high correlation between all features, but that's expected as features are all calculated based on some price or volume change behavior
2. We could use a smaller subset of manually determined, less correlated features. Or, we could manually engineer new features based on commonly used technical indicators (such as the "death cross", where the short term MA (typically 50 days) goes below the long term moving average (typically 200 days)
3. Other methods are out of scope for this work, as you will see later, our model actually performs very well

# Visually inspecting feature importance
fig2 = make_subplots(rows=2, cols=2)

# Adding features by importance for 7 day model
feats_7 = [x for _, x in sorted(zip(feat_importance_7,updated_feats),reverse=True)]
importance_7 = sorted(feat_importance_7,reverse=True)


fig2.add_trace(
    go.Bar(x=feats_7, y=importance_7,name='7 Day'),
    row=1, col=1
)

# Adding features by importance for 30 day model
feats_30 = [x for _, x in sorted(zip(feat_importance_30,updated_feats),reverse=True)]
importance_30 = sorted(feat_importance_30,reverse=True)
fig2.add_trace(
    go.Bar(x=feats_30, y=importance_30,name='30 Day'),
    row=1, col=2
)

# Adding features by importance for 60 day model
feats_60 = [x for _, x in sorted(zip(feat_importance_60,updated_feats),reverse=True)]
importance_60 = sorted(feat_importance_60,reverse=True)
fig2.add_trace(
    go.Bar(x=feats_60, y=importance_60,name='60 Day'),
    row=2, col=1
)

# Adding features by importance for 60 day model
feats_120 = [x for _, x in sorted(zip(feat_importance_120,updated_feats),reverse=True)]
importance_120 = sorted(feat_importance_120,reverse=True)
fig2.add_trace(
    go.Bar(x=feats_120, y=importance_120,name='120 Day'),
    row=2, col=2
)

fig2.update_layout(height=1200, width=1000, title_text="Feature Importance by Target")
fig2.show()

Observations:

1. Every model does a fairly good job of predicting the first data point forecasted in the future, but performance drops off significantly from there
2. Also, our train predictions all have R^2 greater than 99%, which is common for Random Forest models, that can have a tendency to overfit
3. Interesting enough, all models have variation in what features they think are important, and the 7 day model has a greater didstribution of technical features with similar importance

Part 2b: Perform GridSearch to Find Best Model Configuration

GridSearch will be performed to look for optimal model configuration. This can also help limit compute time, by allowing you to find a less compute intense model upfront (e.g. n_estimators = 500 takes substantially more time to train that n_estimators = 100)

def pipeline(X, y):
    """Trains, tests, and evaluates classification model using GridSearch"""
    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
    param_grid = {'n_estimators': [100, 200, 500],
              'max_depth': [5, 10, 15],
              "max_features": ['auto', 'sqrt', 'log2'],
              "criterion": ['mse', 'mae']}
    scorers = {
        'mse': make_scorer(mean_squared_error),
        'max_error' : make_scorer(max_error)
    }

    etc = RandomForestRegressor(random_state=0,n_jobs=-1)
    clf = GridSearchCV(etc, param_grid=param_grid, cv=3, refit='mse',
                       return_train_score=True, scoring=scorers)
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    t = clf.best_params_
    print(t)

    results = pd.DataFrame(clf.cv_results_)
#     results.to_csv('mod_1_gsresults.csv')
    return t, X_train, X_test, y_train, y_test, clf, results

# Determine best params
t, X_train, X_test, y_train, y_test, clf, cv_results = pipeline(train_df[updated_feats], train_df['target_30'])

{'criterion': 'mae', 'max_depth': 5, 'max_features': 'sqrt', 'n_estimators': 100}

Creating Dataframes of model metrics for AAPL

# Savings model performance to file; Used for dashboard
target_list = ['target_7','target_60','target_120']
feat_lists = []
r2_list = []

for item in target_list:

    regr = RandomForestRegressor(max_depth=5,n_estimators=100,criterion='mae',max_features='sqrt',\
                                                 random_state=0,n_jobs=-1)
    regr.fit(train_df[updated_feats],train_df[item])
    
    feat_importance = list(regr.feature_importances_)
    r2 = np.round(regr.score(train_df[updated_feats],train_df[item]),4)
    
    feat_lists.append(feat_importance)
    r2_list.append(r2)

r2_list

[0.9912, 0.9853, 0.9887]

feat_df = pd.DataFrame(data=feat_lists,
                        index=['Target 7','Target 60','Target 120']).transpose()

feat_df['Features'] = updated_feats
# r2 = pd.DataFrame(data=[r2_7,r2_60,r2_120],index=['r2 7','r2 60','r2 120'],columns=['R2 Score'])

feat_df.head()

	Target 7	Target 60	Target 120	Features
0	0.064403	0.046331	0.013881	Low_adj
1	0.094777	0.085231	0.045543	others_cr
2	0.114207	0.087430	0.036689	Adj Close
3	0.108561	0.030517	0.022900	Open_adj
4	0.056222	0.070169	0.060653	volume_vwap

r2_df = pd.DataFrame(data=r2_list,columns=['r2'],index=['Target 7','Target 60','Target 120']).transpose()

# Saving all dfs to file for use in dashboard
feat_df.to_csv('assets/models/tyler_rf_daily_update/feature_importance.csv')
r2_df.to_csv('assets/models/tyler_rf_daily_update/r2_df.csv')
corr.to_csv('assets/models/tyler_rf_daily_update/corr.csv')

Part 3: Using past knowledge to re-train our model

Given we see every first point we predict is much more aligned to actual test data, we will explore an approach that will allow us to re-train the model every day once we have additional info

new_train_df = train_df
test_preds = []
test_dates = []

# Need to remove last 7 observations due to missing price for 7 days in the future
for i in range(len(test_df)-7):

    # Creating next prediction point
    test_point = test_df[updated_feats].iloc[[i]]

    X = new_train_df[updated_feats]
    y = new_train_df['target_7']

    regr = RandomForestRegressor(max_depth=5,n_estimators=100,criterion='mae',max_features='sqrt',random_state=0)

    regr.fit(X, y)

    test_prediction = regr.predict(test_point)[0]
    test_date = test_point.index.values[0]

    test_preds.append(test_prediction)
    test_dates.append(test_date)
    
    # Adding next observation to our training data to simulate re-training a model daily
    new_train_df = pd.concat([new_train_df,test_df.iloc[[i]]])
    
# Adding the final predictions for obersavtions without the target variable
for i in range(len(test_df)-7,len(test_df)):
    
    test_point = test_df[updated_feats].iloc[[i]]
    
    test_prediction = regr.predict(test_point)[0]
    test_date = test_point.index.values[0]
    
    test_preds.append(test_prediction)
    test_dates.append(test_date)

# Creating DF with new preds to allow for easy plotting
new_test_df = pd.DataFrame(data=test_preds,index=test_dates,columns=['7_Day'])

# Plotting our new re-fitted predictions with test data
fig = go.Figure()

fig.add_trace(go.Scatter(x=test_df.index,
                         y=test_df['Close'],
                        line={"color": "#228B22","dash":"dash"},
                        mode="lines",
                        name='Test Actuals'))

fig.add_trace(go.Scatter(x=new_test_df.index+timedelta(days=7),
                         y=new_test_df['7_Day'],
                        line={"color": "red"},
                        mode="lines",
                        name='Test Predictions (7 Day)'))

Observations:

1. As expected, re-training our model daily provides much better accuracy for making predictions into the future, as we are able to incorporate new information for each model build
2. I believe this is true given we are able to minimize the effects of the "random walk" theory commonly attributed to stock market price movements. We are also able to better minimize the effects of exogenous or econmomic factors that actually lead to more concrete changes in stock prices

Part 4: Picking a subset of tickers and creating models for each

In this section, we will make assumptions about tickers and their performance (based on research/tribal knowledge) and build models for each to allow us to start to create a diversified portfolio. We will then iteratively train models for each ticker over different prediction time horizons and portfolio rebalance cycles to compare performance

# Getting all sectors to get largest tickers by volume, as this is a common approach by industry professionals
sectors = list(feature_df.sector.unique())

# Creating function to get largest tickers for a sector
def largest_tickers_by_vol(sector):
    return list(feature_df[feature_df['sector']==sector].groupby('ticker').mean('Volume')['Volume'].nlargest(5).index.values)

tickers = []

for sector in sectors:
    tickers.extend(largest_tickers_by_vol(sector))

tickers[:10]

['PFE', 'BMY', 'MRK', 'GILD', 'BSX', 'GE', 'AAL', 'CSX', 'DAL', 'UAL']

# Setting these manually so I dont have to re-run the rest of the notebook
updated_feats = ['Low_adj','trend_sma_fast','Open_adj','volatility_kch','volatility_bbh','volatility_dch',
 'volume_vwap','trend_ema_slow','trend_ema_fast','others_cr','SMA_15','upperband','trend_sma_slow',
 'volatility_bbm','volatility_kcl','Adj Close','trend_ichimoku_conv','volatility_dcm','volatility_bbl',
 'trend_ichimoku_a']

# Define function to fit a model that predicts price performance over some time horizon
def fit_model(ticker,train_start_dt,train_end_dt,target=7):

    # Creating sample df for ticker only
    sample_df = feature_df[(feature_df['ticker']==ticker)&(feature_df.index>=train_start_dt)&\
                           (feature_df.index<=train_end_dt)]#.fillna(method="ffill")

    # Creating target variable to predict prices X days in future
    sample_df[f'target_{target}'] = sample_df['Close_adj'].shift(-1*target)

    # Dropping 7 most recent dates given there is no prediction
    train_df = sample_df.dropna()

    # Fitting my model on inital train_df
    X = train_df[updated_feats]
    y = train_df[f'target_{target}']

    regr = RandomForestRegressor(max_depth=5,n_estimators=100,criterion='mae',max_features='sqrt',\
                                                 random_state=0,n_jobs=-1)
    regr.fit(X, y)

    # Getting trading days calendar to add prediction to N trading days in future
    nyse = mcal.get_calendar('NYSE')
    train_end_dt_for_cal = datetime.strptime(train_end_dt,"%Y-%m-%d")
    valid_days = list(nyse.valid_days(start_date=train_end_dt_for_cal, end_date=train_end_dt_for_cal+timedelta(days=target+120)))

    # Getting X trading days in future
    test_date = valid_days[target].strftime('%Y-%m-%d')
    test_prediction = regr.predict(sample_df[updated_feats].iloc[[-1]])[0]

    return test_date, test_prediction, regr

# Testing out our function with one model creation for AAPL
fit_model('AAL','2016-01-01','2020-05-28')

('2020-06-08',
 10.301899995803833,
 RandomForestRegressor(criterion='mae', max_depth=5, max_features='sqrt',
                       n_jobs=-1, random_state=0))

# Putting everything above together
# This will assess prices and predictions every 7 days and change positions accordingly

# Setting training dates. Will test on all data after
train_start_dt = '2018-01-01'

# Filtering to new_df just with dates we care about
new_df = feature_df[feature_df.index >=train_start_dt]

# Creating train_df and test_df
split_perc = .8
# Getting index values needed to split df
train_int = int(len(new_df.index.unique())*split_perc)

# Splitting new_df into train_df and test_df
train_df = new_df[new_df.index.isin(new_df.index.unique()[:train_int])]
test_df = new_df[new_df.index.isin(new_df.index.unique()[train_int:])]

train_end_dt = pd.to_datetime(new_df[new_df.index.isin(new_df.index.unique()[:train_int])]\
                              .index.max()).date().strftime('%Y-%m-%d')

# Initializing portfolio
test = port2.portfolio(start_date='2018-01-01', value=100000, end_date='2021-07-30')

ticker_preds = pd.DataFrame(columns=['ticker','pred_date','pred_price','curr_dt','curr_price','earn_ratio'])

# Used for testing and breaking after certain number of loops
iteration = 0

# Setting configuration variables
target = 7
rebal_interval = 7

for i in range(0,len(test_df.index.unique()),rebal_interval):

    # Changing train end date for every iteration, first iteration will be current train_end_dt
    next_train_end_dt = test_df.index.unique()[i].strftime('%Y-%m-%d')
#     next_train_end_dt = (pd.to_datetime(train_end_dt)+timedelta(i)).strftime('%Y-%m-%d')

    # Need to stop early as some ticker price data is missing
    if test_df.index.unique()[i] >= pd.to_datetime('2021-06-22'):
        break

    # Getting last trading day from training data to trade at end of day
    trade_day = new_df[new_df.index<=next_train_end_dt].index[-1].strftime("%Y-%m-%d")

    # Re-fit every ticker model and predict price in the future
    for ticker in tickers:
        
        try: # Try/except is needed for tickers without data

            pred_date, pred_price, model = fit_model(ticker,train_start_dt,next_train_end_dt,target=target)

            # Get current price on last train dt, which is when we would buy stock
            curr_price = test.get_price(date=trade_day,ticker=ticker)
            earn_ratio = pred_price / curr_price - 1

            ticker_preds = ticker_preds.append({'ticker':ticker,'pred_date':pred_date,'pred_price':pred_price,\
                        'curr_dt':next_train_end_dt,'curr_price':curr_price,'earn_ratio':earn_ratio},ignore_index=True)

        except Exception as e:
            print(e)
            print(ticker)
            pass

    # Creating dictinary of stocks and positions to sell
    try:
        sell_dict = create_sell_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.open_positions_df,\
                                                         sell_threshold = -0.05)
        print(f'Current Selected Sells: {sell_dict} for {trade_day}')

        # Selling positions of stocks predicted to decline on first available day of training (EOD)
        
        test.execute_trades(sell_dict,trade_day,'Sell')

    # If no sells exist, then pass
    except Exception as e:
        print(e)
        pass

    # Creating dictinary of stocks and positions to buy
    try:
        buy_dict = create_buy_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.current_cash,\
                                                                                   buy_threshold=0.05)
        print(f'Current Selected Buys: {buy_dict} for {trade_day}')

        # Buying positions of stocks predicted to increase on day of training (EOD)
        test.execute_trades(buy_dict,trade_day,'Buy')

    # If no buying exists, then pass
    except Exception as e:
        print(e)
        pass

    iteration+=1

    print(f'Current Progress: {iteration} out of {int(len(test_df.index.unique())/rebal_interval)}')

    # Adding a break point for testing
#     if iteration >= 10:
#         break

Current Selected Sells: {} for 2020-10-19
Current Selected Buys: {'PEAK': 2552, 'MRO': 1418, 'T': 167, 'WFC': 155, 'KIM': 254, 'MO': 70, 'CCL': 174, 'GILD': 40, 'XOM': 59, 'C': 42, 'HAL': 140, 'INTC': 28, 'MRK': 20, 'KMI': 113} for 2020-10-19
Order to BUY 2552 shares of PEAK validated and approved
Order to BUY 1418 shares of MRO validated and approved
Order to BUY 167 shares of T validated and approved
Order to BUY 155 shares of WFC validated and approved
Order to BUY 254 shares of KIM validated and approved
Order to BUY 70 shares of MO validated and approved
Order to BUY 174 shares of CCL validated and approved
Order to BUY 40 shares of GILD validated and approved
Order to BUY 59 shares of XOM validated and approved
Order to BUY 42 shares of C validated and approved
Order to BUY 140 shares of HAL validated and approved
Order to BUY 28 shares of INTC validated and approved
Order to BUY 20 shares of MRK validated and approved
Order to BUY 113 shares of KMI validated and approved
Trades Executed
Current Progress: 1 out of 25
Current Selected Sells: {} for 2020-10-28
'Quantity'
Current Selected Buys: {'PEAK': 2} for 2020-10-28
Order to BUY 2 shares of PEAK validated and approved
Trades Executed
Current Progress: 2 out of 25
Current Selected Sells: {} for 2020-11-06
'Quantity'
Current Selected Buys: {'PEAK': 2} for 2020-11-06
Order to BUY 2 shares of PEAK validated and approved
Trades Executed
Current Progress: 3 out of 25
Current Selected Sells: {} for 2020-11-17
'Quantity'
Current Selected Buys: {} for 2020-11-17
'Ticker'
Current Progress: 4 out of 25
Current Selected Sells: {} for 2020-11-27
'Quantity'
Current Selected Buys: {} for 2020-11-27
'Ticker'
Current Progress: 5 out of 25
Current Selected Sells: {} for 2020-12-08
'Quantity'
Current Selected Buys: {} for 2020-12-08
'Ticker'
Current Progress: 6 out of 25
Current Selected Sells: {} for 2020-12-17
'Quantity'
Current Selected Buys: {} for 2020-12-17
'Ticker'
Current Progress: 7 out of 25
Current Selected Sells: {} for 2020-12-29
'Quantity'
Current Selected Buys: {} for 2020-12-29
'Ticker'
Current Progress: 8 out of 25
Current Selected Sells: {} for 2021-01-08
'Quantity'
Current Selected Buys: {'PEAK': 1} for 2021-01-08
Order to BUY 1 shares of PEAK validated and approved
Trades Executed
Current Progress: 9 out of 25
Current Selected Sells: {} for 2021-01-20
'Quantity'
Current Selected Buys: {} for 2021-01-20
'Ticker'
Current Progress: 10 out of 25
Current Selected Sells: {} for 2021-01-29
'Quantity'
Current Selected Buys: {} for 2021-01-29
'Ticker'
Current Progress: 11 out of 25
Current Selected Sells: {} for 2021-02-09
'Quantity'
Current Selected Buys: {} for 2021-02-09
'Ticker'
Current Progress: 12 out of 25
Current Selected Sells: {} for 2021-02-19
'Quantity'
Current Selected Buys: {} for 2021-02-19
'Ticker'
Current Progress: 13 out of 25
Current Selected Sells: {} for 2021-03-02
'Quantity'
Current Selected Buys: {} for 2021-03-02
'Ticker'
Current Progress: 14 out of 25
Current Selected Sells: {} for 2021-03-11
'Quantity'
Current Selected Buys: {} for 2021-03-11
'Ticker'
Current Progress: 15 out of 25
Current Selected Sells: {} for 2021-03-22
'Quantity'
Current Selected Buys: {} for 2021-03-22
'Ticker'
Current Progress: 16 out of 25
Current Selected Sells: {} for 2021-03-31
'Quantity'
Current Selected Buys: {} for 2021-03-31
'Ticker'
Current Progress: 17 out of 25
Current Selected Sells: {} for 2021-04-12
'Quantity'
Current Selected Buys: {} for 2021-04-12
'Ticker'
Current Progress: 18 out of 25
Current Selected Sells: {} for 2021-04-21
'Quantity'
Current Selected Buys: {} for 2021-04-21
'Ticker'
Current Progress: 19 out of 25
Current Selected Sells: {} for 2021-04-30
'Quantity'
Current Selected Buys: {} for 2021-04-30
'Ticker'
Current Progress: 20 out of 25
Current Selected Sells: {} for 2021-05-11
'Quantity'
Current Selected Buys: {} for 2021-05-11
'Ticker'
Current Progress: 21 out of 25
Current Selected Sells: {} for 2021-05-20
'Quantity'
Current Selected Buys: {} for 2021-05-20
'Ticker'
Current Progress: 22 out of 25
Current Selected Sells: {} for 2021-06-01
'Quantity'
Current Selected Buys: {} for 2021-06-01
'Ticker'
Current Progress: 23 out of 25
Current Selected Sells: {} for 2021-06-10
'Quantity'
Current Selected Buys: {} for 2021-06-10
'Ticker'
Current Progress: 24 out of 25
Current Selected Sells: {} for 2021-06-21
'Quantity'
Current Selected Buys: {} for 2021-06-21
'Ticker'
Current Progress: 25 out of 25

# track_record has point in time view of how valuable your portfolio is
tr = test.track_record
tr['Date'] = pd.to_datetime(tr['Date'])
tr.tail(10)

	Date	Value	Val_ex_cash
158	2021-06-07	151361.225139	151357.786966
159	2021-06-08	151348.666319	151345.228147
160	2021-06-09	152288.190891	152284.752718
161	2021-06-10	152797.473887	152794.035715
162	2021-06-11	152727.490645	152724.052473
163	2021-06-14	152965.410119	152961.971947
164	2021-06-15	152885.820590	152882.382417
165	2021-06-16	151615.342741	151611.904568
166	2021-06-17	149310.843319	149307.405147
167	2021-06-18	145732.861447	145729.423274

performance_chart(tr, 'spy')

# The risk adjusted metrics below measure well a portfolio compares to a risk-free investment
# Higher ratios mean beter performance
risk_adjusted_metrics(tr, 'spy')

# A variation of above, but the ratio of Return to Volatility for our modeled portfolio is better than the S&P 500
risk_to_ret(tr, 'spy')

# Portfolio is highly weighted towards Real Estate and Energy stocks
snap_dt = tr.Date.max().strftime('%Y-%m-%d')
sector_plot(test.snapshots[f'Positions_{snap_dt}'],test.snapshots[f'cash_{snap_dt}'],date=snap_dt)

# Capital Asset Pricing Model metrics are also favorable; Risk required appears to be equivalent to return expected
capm_res(tr, 'spy')

Observations:

1. Overall the portfolio outperforms the S&P 500 from Oct'20 thru Jun'21
2. The algorithm actually never sells any stock once it buys to begin with, but the mix does well enough that substantial drops do not occur
3. Real Estate is a significant portion of the portfolio, which has performed well as of late, but this could create some diversification risk if we run into recessionary times similar to the Great Recession
4. Common market risk metrics are also favorable for our portfolio over a risk-free investment, which is a common way to assess portfolio health

Preparing and loading data to postgres

1. The output of the portfolio and trading is used to build visualizations in our dashboard
2. This section is used to prepare data for loading, but loading is done in /assets/models/tyler_rf_daily_update/Loading_Model_Data_Postgres.ipynb notebook

# Setting model name to create keys for filtering
model_name = f'RF Reg_target_{target}_rebal_{rebal_interval}_{train_start_dt}'
print(model_name)

# Creating dictionary for open positions (splitting out three separate structures)
open_positions = {}
for key, value in test.snapshots.items():
    
    if key[:4] == 'Posi':
        open_positions[key] = value

# Turning open positions into a dataframe and adding model name
open_positions_df = pd.concat(open_positions, axis=0).reset_index(level=0)\
                            .rename({'level_0':'key'}, axis=1)

open_positions_df['model'] = model_name

# Saving data to file
open_positions_df.to_csv(f'assets/models/tyler_rf_daily_update/Open_Positions_Data/{model_name}.csv')

# Creating dictionary for cash positions (splitting out three separate structures)
cash_positions = {}
for key, value in test.snapshots.items():
    
    if key[:4] == 'cash':
        cash_positions[key] = value
        
# Turning open positions into a dataframe and adding model name
cash_pos_df = pd.DataFrame.from_dict(cash_positions, orient='index').reset_index().rename(columns={'index':'key',0:'cash'})
cash_pos_df['model'] = model_name

# Saving data to file
cash_pos_df.to_csv(f'assets/models/tyler_rf_daily_update/Cash_Positions_Data/{model_name}.csv')

# Creating dictionary for open positions (splitting out three separate structures)
value_positions = {}
for key, value in test.snapshots.items():
    
    if key[:4] == 'val_':
        value_positions[key] = value
        
# Turning value positions into a dataframe and adding model name
val_pos_df = pd.DataFrame.from_dict(value_positions, orient='index').reset_index().rename(columns={'index':'key',0:'value'})
val_pos_df['model'] = model_name

# Saving data to file
val_pos_df.to_csv(f'assets/models/tyler_rf_daily_update/Val_Positions_Data/{model_name}.csv')

# Saving track record to file
tr_df = test.track_record
tr_df['model'] = model_name
tr_df.to_csv(f'assets/models/tyler_rf_daily_update/track_record/{model_name}.csv')

RF Reg_target_7_rebal_7_2018-01-01

Training and Building Portfolios Over Multiple Time Horizons

In this section, we will configure multiple buy/sell strategies based on portfolio rebalance timeframes and prediction timeframes. In the interest of simplifying, we will only run one configuration at a time

# "grid search" configuration for testing different portfolio constructions
gridsearch_config = [(120,60,'2018-01-01')]

# Not Run:

# Stored already:
# (60,30,'2018-01-01') (120,30,'2018-01-01') (120,60,'2018-01-01')

for target, rebal_interval, train_start_dt in gridsearch_config:

    # If it fails, I want it to keep running
    try:
        # Setting training dates. Will test on all data after
        train_start_dt = train_start_dt

        # Filtering to new_df just with dates we care about
        new_df = feature_df[feature_df.index >=train_start_dt]

        # Creating train_df and test_df
        split_perc = .8
        # Getting index values needed to split df
        train_int = int(len(new_df.index.unique())*split_perc)

        # Splitting new_df into train_df and test_df
        train_df = new_df[new_df.index.isin(new_df.index.unique()[:train_int])]
        test_df = new_df[new_df.index.isin(new_df.index.unique()[train_int:])]

        train_end_dt = pd.to_datetime(new_df[new_df.index.isin(new_df.index.unique()[:train_int])]\
                                      .index.max()).date().strftime('%Y-%m-%d')

        # Initializing portfolio
        test = port2.portfolio(start_date=train_start_dt, value=100000, end_date='2021-07-30')

        ticker_preds = pd.DataFrame(columns=['ticker','pred_date','pred_price','curr_dt','curr_price','earn_ratio'])

        # Used for testing and breaking after certain number of loops
        iteration = 0

        for i in range(0,len(test_df.index.unique()),rebal_interval):

            # Changing train end date for every iteration, first iteration will be current train_end_dt
            next_train_end_dt = test_df.index.unique()[i].strftime('%Y-%m-%d')
        #     next_train_end_dt = (pd.to_datetime(train_end_dt)+timedelta(i)).strftime('%Y-%m-%d')

            # Need to stop early as some ticker price data is missing
            if test_df.index.unique()[i] >= pd.to_datetime('2021-06-22'):
                break

            # Getting last trading day from training data to trade at end of day
            trade_day = new_df[new_df.index<=next_train_end_dt].index[-1].strftime("%Y-%m-%d")

            # Re-fit every ticker model and predict price in the future
            for ticker in tickers:

                try: # Try/except is needed for tickers without data

                    pred_date, pred_price, model = fit_model(ticker,train_start_dt,next_train_end_dt,target=target)

                    # Get current price on last train dt, which is when we would buy stock
                    curr_price = test.get_price(date=trade_day,ticker=ticker)
                    earn_ratio = pred_price / curr_price - 1

                    ticker_preds = ticker_preds.append({'ticker':ticker,'pred_date':pred_date,'pred_price':pred_price,\
                                'curr_dt':next_train_end_dt,'curr_price':curr_price,'earn_ratio':earn_ratio},ignore_index=True)

                except Exception as e:
                    print(e)
                    print(ticker)
                    pass

            # Creating dictinary of stocks and positions to sell
            try:
                sell_dict = create_sell_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.open_positions_df,\
                                                                 sell_threshold = -0.05)
                print(f'Current Selected Sells: {sell_dict} for {trade_day}')

                # Selling positions of stocks predicted to decline on first available day of training (EOD)

                test.execute_trades(sell_dict,trade_day,'Sell')

            # If no sells exist, then pass
            except Exception as e:
                print(e)
                pass

            # Creating dictinary of stocks and positions to buy
            try:
                buy_dict = create_buy_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.current_cash,\
                                                                                           buy_threshold=0.05)
                print(f'Current Selected Buys: {buy_dict} for {trade_day}')

                # Buying positions of stocks predicted to increase on day of training (EOD)
                test.execute_trades(buy_dict,trade_day,'Buy')

            # If no buying exists, then pass
            except Exception as e:
                print(e)
                pass

            iteration+=1

            print(f'Current Progress: {iteration} out of {int(len(test_df.index.unique())/rebal_interval)+1}')

            # Adding a break point for testing
        #     if iteration >= 10:
        #         break


    except Exception as e:
        print(e)
        pass

Current Selected Sells: {} for 2020-10-19
Current Selected Buys: {'PEAK': 1765, 'BAC': 175, 'GILD': 68, 'T': 145, 'KIM': 318, 'HAL': 281, 'PPL': 117, 'RF': 259, 'GM': 87, 'F': 311, 'C': 55, 'JPM': 22, 'KO': 44, 'KMI': 179, 'EXC': 49, 'XOM': 60, 'GE': 236, 'BSX': 41, 'OXY': 143, 'MO': 35, 'MRK': 17, 'CCL': 81, 'HST': 97, 'MRO': 260} for 2020-10-19
Order to BUY 1765 shares of PEAK validated and approved
Order to BUY 175 shares of BAC validated and approved
Order to BUY 68 shares of GILD validated and approved
Order to BUY 145 shares of T validated and approved
Order to BUY 318 shares of KIM validated and approved
Order to BUY 281 shares of HAL validated and approved
Order to BUY 117 shares of PPL validated and approved
Order to BUY 259 shares of RF validated and approved
Order to BUY 87 shares of GM validated and approved
Order to BUY 311 shares of F validated and approved
Order to BUY 55 shares of C validated and approved
Order to BUY 22 shares of JPM validated and approved
Order to BUY 44 shares of KO validated and approved
Order to BUY 179 shares of KMI validated and approved
Order to BUY 49 shares of EXC validated and approved
Order to BUY 60 shares of XOM validated and approved
Order to BUY 236 shares of GE validated and approved
Order to BUY 41 shares of BSX validated and approved
Order to BUY 143 shares of OXY validated and approved
Order to BUY 35 shares of MO validated and approved
Order to BUY 17 shares of MRK validated and approved
Order to BUY 81 shares of CCL validated and approved
Order to BUY 97 shares of HST validated and approved
Order to BUY 260 shares of MRO validated and approved
Trades Executed
Current Progress: 1 out of 3
Current Selected Sells: {} for 2021-01-14
'Quantity'
Current Selected Buys: {'PEAK': 9, 'KIM': 1} for 2021-01-14
Order to BUY 9 shares of PEAK validated and approved
Order to BUY 1 shares of KIM validated and approved
Trades Executed
Current Progress: 2 out of 3
Current Selected Sells: {} for 2021-04-13
'Quantity'
Current Selected Buys: {'PEAK': 2} for 2021-04-13
Order to BUY 2 shares of PEAK validated and approved
Trades Executed
Current Progress: 3 out of 3

# Again preparing portfolio data for dashboard
# Setting model name to create keys for filtering postgres tables
model_name = f'RF Reg_target_{target}_rebal_{rebal_interval}_{train_start_dt}'
print(model_name)

# Creating dictionary for open positions (splitting out three separate structures)
open_positions = {}
for key, value in test.snapshots.items():
    
    if key[:4] == 'Posi':
        open_positions[key] = value

# Turning open positions into a dataframe and adding model name
open_positions_df = pd.concat(open_positions, axis=0).reset_index(level=0)\
                            .rename({'level_0':'key'}, axis=1)

open_positions_df['model'] = model_name

# Saving data to file
open_positions_df.to_csv(f'assets/models/tyler_rf_daily_update/Open_Positions_Data/{model_name}.csv')

# Creating dictionary for cash positions (splitting out three separate structures)
cash_positions = {}
for key, value in test.snapshots.items():
    
    if key[:4] == 'cash':
        cash_positions[key] = value
        
# Turning open positions into a dataframe and adding model name
cash_pos_df = pd.DataFrame.from_dict(cash_positions, orient='index').reset_index().rename(columns={'index':'key',0:'cash'})
cash_pos_df['model'] = model_name

# Saving data to file
cash_pos_df.to_csv(f'assets/models/tyler_rf_daily_update/Cash_Positions_Data/{model_name}.csv')

# Creating dictionary for open positions (splitting out three separate structures)
value_positions = {}
for key, value in test.snapshots.items():
    
    if key[:4] == 'val_':
        value_positions[key] = value
        
# Turning value positions into a dataframe and adding model name
val_pos_df = pd.DataFrame.from_dict(value_positions, orient='index').reset_index().rename(columns={'index':'key',0:'value'})
val_pos_df['model'] = model_name

# Saving data to file
val_pos_df.to_csv(f'assets/models/tyler_rf_daily_update/Val_Positions_Data/{model_name}.csv')

# Saving track record to file
tr_df = test.track_record
tr_df['model'] = model_name
tr_df.to_csv(f'assets/models/tyler_rf_daily_update/track_record/{model_name}.csv')

RF Reg_target_120_rebal_60_2018-01-01

Part 5: Backtesting

In this next section, we will work to backtest a couple high performing portfolios, as uncovered from the dashboard. This approach involves finding a time of greater losses (or stress) for the entire market, and seeing how well the strategy does. Given the model mostly runs during a time of positive market movement, we haven't ensured it can handle downturns. For the stressed scenario, we will use the time around the start of the COVID-19 pandemic, where losses were "exaggerated"

# Creating a "spoofed" track record to get market performance in the past
temp_track = feature_df[feature_df['ticker']=='AAPL'][['Close','Adj Close']].reset_index()\
                                    .rename({'Close':'Value','Adj Close':'Val_ex_cash'},axis=1)

fig = performance_chart(temp_track, 'spy')

fig.update_layout(title= dict(text='Performance Chart (AAPL vs S&P 500 (spy))', font = dict(size = 20, color = 'black'), x = 0.5, y = 0.96))

fig.update_yaxes(range=[-.2, 2])

# Adding box around target test range
fig.add_trace(go.Scatter(
    x=['2020-01-01'],
    y=[-.07],
    text=["Test Range for Backtest"],
    mode="text"
))

fig.add_shape(type="rect",
    x0='2019-08-01', y0=0, x1='2020-06-01', y1=.7,
    line=dict(color="RoyalBlue"),
)

fig

# Setting training dates. Will test on all data after
train_start_dt = '2016-01-01'
data_end_dt = '2021-03-30'

# Filtering to new_df just with dates we care about
new_df = feature_df[(feature_df.index >=train_start_dt)&(feature_df.index <=data_end_dt)]

# Creating train_df and test_df
split_perc = .9
# Getting index values needed to split df
train_int = int(len(new_df.index.unique())*split_perc)

# Splitting new_df into train_df and test_df
train_df = new_df[new_df.index.isin(new_df.index.unique()[:train_int])]
test_df = new_df[new_df.index.isin(new_df.index.unique()[train_int:])]

train_end_dt = pd.to_datetime(new_df[new_df.index.isin(new_df.index.unique()[:train_int])]\
                              .index.max()).date().strftime('%Y-%m-%d')

# Initializing portfolio
test = port2.portfolio(start_date=train_start_dt, value=100000, end_date=data_end_dt)

ticker_preds = pd.DataFrame(columns=['ticker','pred_date','pred_price','curr_dt','curr_price','earn_ratio'])

# Used for testing and breaking after certain number of loops
iteration = 0

# Setting configuration variables
target = 120
rebal_interval = 60

for i in range(0,len(test_df.index.unique()),rebal_interval):

    # Changing train end date for every iteration, first iteration will be current train_end_dt
    next_train_end_dt = test_df.index.unique()[i].strftime('%Y-%m-%d')
#     next_train_end_dt = (pd.to_datetime(train_end_dt)+timedelta(i)).strftime('%Y-%m-%d')

    # Need to stop early as some ticker price data is missing
    if test_df.index.unique()[i] >= pd.to_datetime('2021-06-22'):
        break

    # Getting last trading day from training data to trade at end of day
    trade_day = new_df[new_df.index<=next_train_end_dt].index[-1].strftime("%Y-%m-%d")

    # Re-fit every ticker model and predict price in the future
    for ticker in tickers:
        
        try: # Try/except is needed for tickers without data

            pred_date, pred_price, model = fit_model(ticker,train_start_dt,next_train_end_dt,target=target)

            # Get current price on last train dt, which is when we would buy stock
            curr_price = test.get_price(date=trade_day,ticker=ticker)
            earn_ratio = pred_price / curr_price - 1

            ticker_preds = ticker_preds.append({'ticker':ticker,'pred_date':pred_date,'pred_price':pred_price,\
                        'curr_dt':next_train_end_dt,'curr_price':curr_price,'earn_ratio':earn_ratio},ignore_index=True)

        except Exception as e:
            print(e)
            print(ticker)
            pass

    # Creating dictinary of stocks and positions to sell
    try:
        sell_dict = create_sell_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.open_positions_df,\
                                                         sell_threshold = -0.05)
        print(f'Current Selected Sells: {sell_dict} for {trade_day}')

        # Selling positions of stocks predicted to decline on first available day of training (EOD)
        
        test.execute_trades(sell_dict,trade_day,'Sell')

    # If no sells exist, then pass
    except Exception as e:
        print(e)
        pass

    # Creating dictinary of stocks and positions to buy
    try:
        buy_dict = create_buy_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.current_cash,\
                                                                                   buy_threshold=0.05)
        print(f'Current Selected Buys: {buy_dict} for {trade_day}')

        # Buying positions of stocks predicted to increase on day of training (EOD)
        test.execute_trades(buy_dict,trade_day,'Buy')

    # If no buying exists, then pass
    except Exception as e:
        print(e)
        pass

    iteration+=1

    print(f'Current Progress: {iteration} out of {int(len(test_df.index.unique())/rebal_interval)+1}')

    # Adding a break point for testing
#     if iteration >= 10:
#         break

Current Selected Sells: {} for 2020-09-21
Current Selected Buys: {'PEAK': 1108, 'KMI': 472, 'GE': 848, 'DAL': 179, 'KIM': 445, 'HST': 454, 'GM': 109, 'T': 116, 'RF': 276, 'EXC': 88, 'PPL': 117, 'SO': 55, 'BAC': 108, 'HAL': 188, 'MO': 64, 'JPM': 24, 'C': 48, 'UAL': 50, 'DD': 28, 'CCL': 99, 'KO': 27, 'XOM': 32, 'GILD': 15, 'PLD': 9, 'PFE': 26, 'MSFT': 3, 'AMD': 10, 'NEM': 12, 'OXY': 65, 'BMY': 11, 'AAL': 47, 'AAPL': 5} for 2020-09-21
Order to BUY 1108 shares of PEAK validated and approved
Order to BUY 472 shares of KMI validated and approved
Order to BUY 848 shares of GE validated and approved
Order to BUY 179 shares of DAL validated and approved
Order to BUY 445 shares of KIM validated and approved
Order to BUY 454 shares of HST validated and approved
Order to BUY 109 shares of GM validated and approved
Order to BUY 116 shares of T validated and approved
Order to BUY 276 shares of RF validated and approved
Order to BUY 88 shares of EXC validated and approved
Order to BUY 117 shares of PPL validated and approved
Order to BUY 55 shares of SO validated and approved
Order to BUY 108 shares of BAC validated and approved
Order to BUY 188 shares of HAL validated and approved
Order to BUY 64 shares of MO validated and approved
Order to BUY 24 shares of JPM validated and approved
Order to BUY 48 shares of C validated and approved
Order to BUY 50 shares of UAL validated and approved
Order to BUY 28 shares of DD validated and approved
Order to BUY 99 shares of CCL validated and approved
Order to BUY 27 shares of KO validated and approved
Order to BUY 32 shares of XOM validated and approved
Order to BUY 15 shares of GILD validated and approved
Order to BUY 9 shares of PLD validated and approved
Order to BUY 26 shares of PFE validated and approved
Order to BUY 3 shares of MSFT validated and approved
Order to BUY 10 shares of AMD validated and approved
Order to BUY 12 shares of NEM validated and approved
Order to BUY 65 shares of OXY validated and approved
Order to BUY 11 shares of BMY validated and approved
Order to BUY 47 shares of AAL validated and approved
Order to BUY 5 shares of AAPL validated and approved
Trades Executed
Current Progress: 1 out of 3
Current Selected Sells: {} for 2020-12-15
'Quantity'
Current Selected Buys: {'PEAK': 13, 'UAL': 1, 'KIM': 2, 'PPL': 1, 'HST': 2, 'MRO': 2, 'KMI': 1} for 2020-12-15
Order to BUY 13 shares of PEAK validated and approved
Order to BUY 1 shares of UAL validated and approved
Order to BUY 2 shares of KIM validated and approved
Order to BUY 1 shares of PPL validated and approved
Order to BUY 2 shares of HST validated and approved
Order to BUY 2 shares of MRO validated and approved
Order to BUY 1 shares of KMI validated and approved
Trades Executed
Current Progress: 2 out of 3
Current Selected Sells: {} for 2021-03-15
'Quantity'
Current Selected Buys: {'PEAK': 5, 'MRO': 1} for 2021-03-15
Order to BUY 5 shares of PEAK validated and approved
Order to BUY 1 shares of MRO validated and approved
Trades Executed
Current Progress: 3 out of 3

# Pulling track record and comparing performance of the portfolio to S&P 500
# Since chart is ROI, overall shape will not be the same, as below represents an investment later in time
tr = test.track_record
tr['Date'] = pd.to_datetime(tr['Date'])

fig2 = performance_chart(tr, 'spy')

fig2.update_layout(title= dict(text='Performance Chart (Stressed Market Scenario)',\
                              font = dict(size = 20, color = 'black'), x = 0.5, y = 0.96))

fig2

snap_dt = tr.Date.max().strftime('%Y-%m-%d')
sector_plot(test.snapshots[f'Positions_{snap_dt}'],test.snapshots[f'cash_{snap_dt}'],date=snap_dt)

# Initializing portfolio for another model
test = port2.portfolio(start_date=train_start_dt, value=100000, end_date=data_end_dt)

ticker_preds = pd.DataFrame(columns=['ticker','pred_date','pred_price','curr_dt','curr_price','earn_ratio'])

# Used for testing and breaking after certain number of loops
iteration = 0

# Setting configuration variables
target = 7
rebal_interval = 7

for i in range(0,len(test_df.index.unique()),rebal_interval):

    # Changing train end date for every iteration, first iteration will be current train_end_dt
    next_train_end_dt = test_df.index.unique()[i].strftime('%Y-%m-%d')
#     next_train_end_dt = (pd.to_datetime(train_end_dt)+timedelta(i)).strftime('%Y-%m-%d')

    # Need to stop early as some ticker price data is missing
    if test_df.index.unique()[i] >= pd.to_datetime('2021-06-22'):
        break

    # Getting last trading day from training data to trade at end of day
    trade_day = new_df[new_df.index<=next_train_end_dt].index[-1].strftime("%Y-%m-%d")

    # Re-fit every ticker model and predict price in the future
    for ticker in tickers:
        
        try: # Try/except is needed for tickers without data

            pred_date, pred_price, model = fit_model(ticker,train_start_dt,next_train_end_dt,target=target)

            # Get current price on last train dt, which is when we would buy stock
            curr_price = test.get_price(date=trade_day,ticker=ticker)
            earn_ratio = pred_price / curr_price - 1

            ticker_preds = ticker_preds.append({'ticker':ticker,'pred_date':pred_date,'pred_price':pred_price,\
                        'curr_dt':next_train_end_dt,'curr_price':curr_price,'earn_ratio':earn_ratio},ignore_index=True)

        except Exception as e:
            print(e)
            print(ticker)
            pass

    # Creating dictinary of stocks and positions to sell
    try:
        sell_dict = create_sell_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.open_positions_df,\
                                                         sell_threshold = -0.05)
        print(f'Current Selected Sells: {sell_dict} for {trade_day}')

        # Selling positions of stocks predicted to decline on first available day of training (EOD)
        
        test.execute_trades(sell_dict,trade_day,'Sell')

    # If no sells exist, then pass
    except Exception as e:
        print(e)
        pass

    # Creating dictinary of stocks and positions to buy
    try:
        buy_dict = create_buy_dict(ticker_preds[ticker_preds['pred_date']==pred_date],test.current_cash,\
                                                                                   buy_threshold=0.05)
        print(f'Current Selected Buys: {buy_dict} for {trade_day}')

        # Buying positions of stocks predicted to increase on day of training (EOD)
        test.execute_trades(buy_dict,trade_day,'Buy')

    # If no buying exists, then pass
    except Exception as e:
        print(e)
        pass

    iteration+=1

    print(f'Current Progress: {iteration} out of {int(len(test_df.index.unique())/rebal_interval)+1}')

    # Adding a break point for testing
#     if iteration >= 10:
#         break

Current Selected Sells: {} for 2020-09-21
Current Selected Buys: {'PEAK': 1817, 'KMI': 362, 'XOM': 109, 'MO': 86, 'OXY': 261, 'MRO': 633, 'T': 103, 'EBAY': 57, 'PPL': 104, 'EXC': 67, 'C': 47, 'GILD': 30, 'FB': 6, 'KIM': 149, 'AAPL': 15, 'HAL': 122, 'KR': 47, 'JPM': 16, 'AMD': 18, 'HST': 115, 'KO': 23, 'GM': 37, 'BMY': 19, 'SO': 21, 'MSFT': 5, 'NEM': 17, 'BAC': 40, 'INTC': 19, 'DOW': 20} for 2020-09-21
Order to BUY 1817 shares of PEAK validated and approved
Order to BUY 362 shares of KMI validated and approved
Order to BUY 109 shares of XOM validated and approved
Order to BUY 86 shares of MO validated and approved
Order to BUY 261 shares of OXY validated and approved
Order to BUY 633 shares of MRO validated and approved
Order to BUY 103 shares of T validated and approved
Order to BUY 57 shares of EBAY validated and approved
Order to BUY 104 shares of PPL validated and approved
Order to BUY 67 shares of EXC validated and approved
Order to BUY 47 shares of C validated and approved
Order to BUY 30 shares of GILD validated and approved
Order to BUY 6 shares of FB validated and approved
Order to BUY 149 shares of KIM validated and approved
Order to BUY 15 shares of AAPL validated and approved
Order to BUY 122 shares of HAL validated and approved
Order to BUY 47 shares of KR validated and approved
Order to BUY 16 shares of JPM validated and approved
Order to BUY 18 shares of AMD validated and approved
Order to BUY 115 shares of HST validated and approved
Order to BUY 23 shares of KO validated and approved
Order to BUY 37 shares of GM validated and approved
Order to BUY 19 shares of BMY validated and approved
Order to BUY 21 shares of SO validated and approved
Order to BUY 5 shares of MSFT validated and approved
Order to BUY 17 shares of NEM validated and approved
Order to BUY 40 shares of BAC validated and approved
Order to BUY 19 shares of INTC validated and approved
Order to BUY 20 shares of DOW validated and approved
Trades Executed
Current Progress: 1 out of 19
Current Selected Sells: {} for 2020-09-30
'Quantity'
Current Selected Buys: {'PEAK': 18, 'XOM': 1, 'KMI': 3, 'T': 1, 'HAL': 2, 'MRO': 5, 'OXY': 1, 'KIM': 1, 'GE': 1} for 2020-09-30
Order to BUY 18 shares of PEAK validated and approved
Order to BUY 1 shares of XOM validated and approved
Order to BUY 3 shares of KMI validated and approved
Order to BUY 1 shares of T validated and approved
Order to BUY 2 shares of HAL validated and approved
Order to BUY 5 shares of MRO validated and approved
Order to BUY 1 shares of OXY validated and approved
Order to BUY 1 shares of KIM validated and approved
Order to BUY 1 shares of GE validated and approved
Trades Executed
Current Progress: 2 out of 19
Current Selected Sells: {} for 2020-10-09
'Quantity'
Current Selected Buys: {'PEAK': 5, 'MRO': 2} for 2020-10-09
Order to BUY 5 shares of PEAK validated and approved
Order to BUY 2 shares of MRO validated and approved
Trades Executed
Current Progress: 3 out of 19
Current Selected Sells: {} for 2020-10-20
'Quantity'
Current Selected Buys: {'PEAK': 1} for 2020-10-20
Order to BUY 1 shares of PEAK validated and approved
Trades Executed
Current Progress: 4 out of 19
Current Selected Sells: {} for 2020-10-29
'Quantity'
Current Selected Buys: {} for 2020-10-29
'Ticker'
Current Progress: 5 out of 19
Current Selected Sells: {} for 2020-11-09
'Quantity'
Current Selected Buys: {'PEAK': 1} for 2020-11-09
Order to BUY 1 shares of PEAK validated and approved
Trades Executed
Current Progress: 6 out of 19
Current Selected Sells: {} for 2020-11-18
'Quantity'
Current Selected Buys: {} for 2020-11-18
'Ticker'
Current Progress: 7 out of 19
Current Selected Sells: {} for 2020-11-30
'Quantity'
Current Selected Buys: {} for 2020-11-30
'Ticker'
Current Progress: 8 out of 19
Current Selected Sells: {} for 2020-12-09
'Quantity'
Current Selected Buys: {} for 2020-12-09
'Ticker'
Current Progress: 9 out of 19
Current Selected Sells: {} for 2020-12-18
'Quantity'
Current Selected Buys: {} for 2020-12-18
'Ticker'
Current Progress: 10 out of 19
Current Selected Sells: {} for 2020-12-30
'Quantity'
Current Selected Buys: {} for 2020-12-30
'Ticker'
Current Progress: 11 out of 19
Current Selected Sells: {} for 2021-01-11
'Quantity'
Current Selected Buys: {} for 2021-01-11
'Ticker'
Current Progress: 12 out of 19
Current Selected Sells: {} for 2021-01-21
'Quantity'
Current Selected Buys: {} for 2021-01-21
'Ticker'
Current Progress: 13 out of 19
Current Selected Sells: {} for 2021-02-01
'Quantity'
Current Selected Buys: {} for 2021-02-01
'Ticker'
Current Progress: 14 out of 19
Current Selected Sells: {} for 2021-02-10
'Quantity'
Current Selected Buys: {} for 2021-02-10
'Ticker'
Current Progress: 15 out of 19
Current Selected Sells: {} for 2021-02-22
'Quantity'
Current Selected Buys: {} for 2021-02-22
'Ticker'
Current Progress: 16 out of 19
Current Selected Sells: {} for 2021-03-03
'Quantity'
Current Selected Buys: {} for 2021-03-03
'Ticker'
Current Progress: 17 out of 19
Current Selected Sells: {} for 2021-03-12
'Quantity'
Current Selected Buys: {} for 2021-03-12
'Ticker'
Current Progress: 18 out of 19
Current Selected Sells: {} for 2021-03-23
'Quantity'
Current Selected Buys: {} for 2021-03-23
'Ticker'
Current Progress: 19 out of 19

# Pulling track record and comparing performance of the portfolio to S&P 500
# Since chart is ROI, overall shape will not be the same, as below represents an investment later in time
tr = test.track_record
tr['Date'] = pd.to_datetime(tr['Date'])

fig2 = performance_chart(tr, 'spy')

fig2.update_layout(title= dict(text='Performance Chart (Stressed Market Scenario)',\
                              font = dict(size = 20, color = 'black'), x = 0.5, y = 0.96))

fig2

snap_dt = tr.Date.max().strftime('%Y-%m-%d')
sector_plot(test.snapshots[f'Positions_{snap_dt}'],test.snapshots[f'cash_{snap_dt}'],date=snap_dt)

Observations:

1. Even during a stressed scenario, our longer term portfolio, predict 120 days in the future and re-balance every 60 days, is at parity with the overall market. This is most likely due to not aggressively selling during stressed scenarios
2. One scenario missing that may be helpful to do another backtest on would be a longer period of stress, such as the Great Recession. Also, portfolio sector weights play a factor and can change performance in different types of stress scenarios. Both Energy (36%) and Real Estate (11%) have all experienced specific stressed scenarios in the past
3. The 7 and 7 model also does a great job of tracking the market during a downturn, which leads me to believe that realized gains during great times would not be lost during bad

import plotly
plotly.offline.init_notebook_mode()